Gesture based User Interface Supporting ^ 
Preexisting Symbols 
Inventors: David L. Marvit et al. 
Attorney Docket No. 073338.0193 
Filed: March 23, 2004 
Sheet 1 of 1 1 

1/11 



14- 



10- 



INPUT 




DISPLAY 












PROCESSOR 



16 



MOTION 
DETECTOR 



22 



20 



•12 



18 







APPLICATIONS 




21 -S 
\ 19 




DATABASES 









COMMUNICATIONS INTERFACE 



FIG. 1 



X-AXIS 
ACCELEROMETER 



Y-AXIS 
ACCELEROMETER 



Z-AXIS 
ACCELEROMETER 



24c 



" 26a-x 


CAMERA 


28a -v 


GYRO 


30 a -v. 


RANGEFINDER 


.^24a 












26b-v 












CAMERA 


28b^. 


GYRO 


30b^ 


RANGEFINDER 


^-24b 
























26c-^ 


CAMERA 


\ 28c^ 


GYRO 


300^ 


RANGEFINDER 



32- 



PR0CESS0R 



FIG. 2 



MOTION DETECTOR 



22 




23a- 
23b- 
23c- 



25a- 



25b- 



25c- 



Gesture Based User Interface Supporting 
Preexisting Symbols 
Inventors: David L. Marvit et al. 
Attorney Docket No. 073338.0193 
Filed: March 23, 2004 
Sheet 2 of 1 1 



X-AXIS ACCELEROMETER 
RAW DATA 



Y-AXIS ACCELEROMETER 
RAW DATA 



Z-AXIS ACCELEROMETER 
RAW DATA 



CAMERA RAW DATA 



CAMERA RAW DATA 



CAMERA RAW DATA 



34 



2/11 



32 



PROCESSOR 



TRANSLATION X 



TRANSLATION Y 



TRANSLATION Z 



ROTATION X 



ROTATION Y 



ROTATION Z 



GYRO 
RAW DATA 



GYRO 
RAW DATA 



GYRO 
RAW DATA 



RANGEFINDER 
RAW DATA 



RANGEFINDER 
RAW DATA 



RANGEFINDER 
RAW DATA 



FIG. 3 



27a 



27b 



27c 



-29a 
29b 
•29c 




PROCESSOR 



DEVICE BEHAVIOR 



41 



44^] 

I 

a 



42- 



46 48 



■50 



FIG. 5 



Gesture Based User Interlace supporting ^ 
Preexisting Symbols 
Inventors: David L. Marvit et al. 
Attorney Docket No. 073338.0193 
Filed: March 23, 2004 
Sheet 3 of 1 1 

"3/11 " 



X-ACCELERATION 
DATA 



I 



64- 



Y-ACCELERATION 
DATA 



66 



Z-ACCELERATION 
DATA 

1 



62C 



J 



60 



AUGMENT X 

7 I 



68a 



"^s. AXIS? 






AUGMENT Y 


| AUGMENT Z 






S 

68c 


if ' \ 







70- 
72- 



PR0CESS 



DEVICE BEHAVIOR 



J USER 1 
' PREFERENCES ' 

69 



FIG. 6 



80 



( START ) 



82a -v DETECT X-AXIS 
ACCELERATION A 

1 



82b- 



FIG. 7 



DETECT Y-AXIS 
ACCELERATION A 



DETECT Z-AXIS 
ACCELERATION A 




Gesture Based User Interface Supporting^ 
Preexisting Symbols 
Inventors: David L. Marvit et al. 
Attorney Docket No. 073338.0193 
Filed: March 23, 2004 
Sheet 4 of 1 1 

4/11 



FIG. 8 



92 

i 



94 



96 



dr. 



98 



ri 
i 

+ 

i 



— r| 



ul- 

J 



f . 

I 



4- L 1- 



i f 



100- 



102- 



104- 



106- 



108- 



CsfMT) FIG. 9 



MOVE DEVICE TO RIGHT 



DISENGAGE MOTION 
SENSITIVITY 



MOVE DEVICE TO LEFT 



REENGAGE MOTION 
SENSITIVITY 



MOVE DEVICE TO RIGHT 




NEED FURTHER 
MOVEMENT? 



YES 



Gesture Based User Interface Supporting^ 
Preexisting Symbols 
Inventors: David L. Marvit et al. 
Attorney Docket No. 073338.0193 
Filed: March 23, 2004 
Sheet 5 of 1 1 



5/11 




FIG. 10A 




135 




DOWN 



136 



LEFT 



137 

L 



e 3» 



RIGHT 



138 




IN 




OUT 



139 



FIG. 10B 



142 



140 



A 


B 


C 


D 


E 


F | 


G 


H 


I 


J 


K 


L 


M 


N 


o 


P 



B1 



B4 



B7 



144 

_Z 



B2 



B5 



B8 



if 



B6 



B9 



146 

jL 



B2a 


B2b 


B2c 


B2d 


B2e 


B2f 


B2g 


B2h 


B2I 



FIG. 11 



Gesture Based User Intertace supporting ^ 
Preexisting Symbols 
Inventors: David L. Marvit et al. 
Attorney Docket No. 073338.0193 
Filed: March 23, 2004 
Sheet 6 of 11 

" &/ ir ~ 



147 



148 



A 


B 


C 
1 


1 


E 


F 


G 


H 


I 


J 


K 


L 


M 


N 


0 


P 



150 
148 



149 



A 

< 


B 


C 


D 


E 


> 

F 


G 


H 


1 


J 


K 


L 


M 


N 


0 


P 



151 



148 



152 

A 



A 


B 


C 


D 


E 


F 


G 


H 


I 

1 


J 

i 


K 


L 


m( 


N 


0 


P 



153 



154 



A 


B 


C 


D 


E 


F 


G 


H 


I 


J 


K 


L 


M 


N 




P 



148 



FIG. 12A 



FIG. 12B I 



A 
B 
C 
D 
E 
F 





160 

s 








162 
/ 






































164 
t 





































161 



B2 
— — « 


B3 


02^ 


C3 



159 



163 



B5 
« 


B6 

> 




C6 



159 



165 



E3 


E4 

i 




F4 



159 



Gesture Based User Interface Supporting^ 
Preexisting Symbols 
Inventors: David L. Marvit et al. 
Attorney Docket No. 073338.0193 
Filed: March 23, 2004 
Sheet 7 of 1 1 

7/11 



170 



172- 

174- 
176- 

178- 



RECEIVE RAW 
MOTION DATA 



PROCESS 



CHECK DEVICE STATE 



ANALYZE MOTION 
DETECTOR OUTPUT 




HANDHELD 
DEVICE 




mCommerce 
APPLICATION 

~7 

202 



200 



204 



AUTHENTICATOR 



FIG. 14 



220 



•222 



HANDHELD DEVICE 



REMOTE DEVICE 



FIG. 15 



•10 




COMMUNICATION 

NETWORK X"-226 



-224 



Gesture Based User Interface Supporting^ 
Preexisting Symbols 
Inventors: David L. Marvit et al. 
Attorney Docket No. 073338.0193 
Filed: March 23, 2004 
Sheet 8 of 1 1 

8/11 



236 



232- 
234- 



RECEIVE RAW MOTION DATA 



230 



PROCESS 



DETERMINE MOTION AND ORIENTATION 



r ___i___ n r .__l_______l. , 

I ROTATING | | TRANSLATING | | ORIENTED AT | 

' AROUND Z-AXIS 1 ' ALONG X-AXIS 1 1 a,0,u I 

..-_ r ____., . v , 

237a 237b 237c 



o o o I STILL 



237n 



DETERMINE ENVIRONMENT BASED ON MOTION AND ORIENTATION 



I FACE DOWN | | 

' ON TABLE I I 

-7— J '"7" 

239a 239b 



FALLING 



I I 



ON TRAIN ] o o o j 



"V" 

239c 



HELD IN | 
HAND • 

239n 



MAP ENVIRONMENT TO BEHAVIOR 



— _ 

ENGAGE i i POWER DOWN CHIPS i i INCREASE MOTION i 

MUTE I I TO SURVIVE IMPACT I ° ° ° I ACTIVATION THRESHOLD ' 
— p 1 i ^ 1 i ^ i 



241a 



241b 



241 n 



242- 



BEHAVE 



240 



FIG. 16 



Gesture Based User interlace Supporting ^ 
Preexisting Symbols 
Inventors: David L. Marvit et al. 
Attorney Docket No. 073338.0193 
Filed: March 23, 2004 
Sheet 9 of 11 

y/u 



250 

! V j 



N 



V 256 

\ 
/ 

I 

/ 



10- 



A 



i 
i 



i |^252 
I i 



io-fll- 



\ 



I 

258^j / 



254 



10 



y\ |^260 



272 




274- 

276- 
278- 



282- 
286- 



RECEIVE RAW 
MOTION DATA 



PROCESS 



MAP MOTION 
TO GESTURE 



"0" 



MAP GESTURE 
TO FUNCTION 



BEHAVE 



280 



GESTURE 
DATABASE 



FUNCTION 
DATABASE 



284 



330 



RECEIVE USER 
INDICATION 
REGARDING 
GESTURE CREATION 



RECEIVE RAW 
MOTION DATA 



PROCESS 



STORE MOTION 
AS GESTURE 



RECEIVE 
FUNCTION MAPPING 
INFORMATION 



STORE 
FUNCTION MAPPING 
INFORMATION 



-332 

-334 
-336 

-338 

-340 



-342 



FIG. 18 



FIG. 21 



Gesture Based User Interface Supporting 
Preexisting Symbols 
Inventors: David L. Marvit et al. 
Attorney Docket No. 073338.0193 
Filed: March 23, 2004 
Sheet 10 of 11 

10/11 



292- 
294- 
296- 



FIG. 19 



RECEIVE RAW 
MOTION DATA 



PROCESS 



MAP MOTION 
TO GESTURE 




290 



1 


APPLICATION 1 


APPLICATION 2 


APPLICATION 3 


APPLICATION 4 

r 


FUNCTION 1 




FUNCTION 2 




FUNCTION 3 




FUNCTION 4 




/ 
300a 




/ 

300b 






s 

300c 






s 

300d 





312-v 



314- 



316-v 



RECEIVE RAW 
MOTION DATA 



PROCESS 



MAP MOTION 
TO GESTURE 



310 



FIG. 20 






USER 1 


i 


USER 2 




USER 3 




USER 4 


FUNCTION 1 




FUNCTION 2 




FUNCTION 3 




FUNCTION 4 






/ 






\ 




s 


320a 


320b 




320c 




320d 



Gesture Based User Interface Supporting > 
Preexisting Symbols 
Inventors: David L. Marvit et al. 
Attorney Docket No. 073338.0193 
Filed: March 23, 2004 
Sheet 11 of 11 

11/11 



s 

/ 

ZZ71 



10- 



\ 



350 



10- 



v 



352 



N 




354 



FIG. 22 



370 
\ 



372- 
374- 
376- 



RECEIVE RAW 
MOTION DATA 



PROCESS 



MAP MOTION 
TO GESTURE 



384- 



382- 



MAP GESTURE 
TO FUNCTION 



FUNCTION 1 



~7" 

392a 



378 



USER 
SETTINGS 
DATABASE 



GESTURE 
DATABASE 



FUNCTION 
MAPPING 
DATABASE 



386 



CONTEXT 
388 



FUNCTION 2 

~7 

392b 



FUNCTION 3 

V~ 

392c 



USER 1 

• PRECISION 

• NOISE 

• USER-CREATED 
GESTURES 



USER 2 



ENVIRONMENTAL MODEL 



USER 1 
USER MAPPING 
USER-CREATED 
FUNCTIONS (MACROS, 
PHONE NUMBERS,...) 



USER 2 



ENVIRONMENTAL MODEL 



APPLICATION IN FOCUS 



DEVICE STATE 



FIG. 23 



-379 



-380 



387 

USER 
" IDENTITIES 



381 



•389 
-390 
-391 



